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DESCRIPTION 



SIGNAL PROCESSING APPARATUS AND METHOD, RECORDING 
MEDIUM, AND PROGRAM 

TECHNICAL FIELD 

The present invention relates to a signal processing apparatus 
and a method, a recording medium, and a program, more particularly to a 
signal processing apparatus and a method, a recording medium, and a 
program which can be suitably used when emphasizing a texture of the 
inside surrounded by edges without emphasizing the edges too much in 
an image, for example. 

BACKGROUND ART 

Conventionally, as for the video camera, a contrast emphasis 
method by way of gradation change, a high frequency component 
emphasis method of emphasizing contrast of a high frequency component 
in an image, etc. have been proposed as a method of improving contrast (a 
difference between brightness and darkness) and a degree of sharpness 
(bordering precision) of an image taken by imaging devices, such as CCD 
(Charge Coupled Device), CMOS (Complementary Metal-Oxide 
Semiconductor), etc. 

As the contrast emphasis method, a tone curve adjustment in 
which a pixel level for each pixel of an image is converted with a function 
(hereafter referred to as a level conversion function) having a 
predetermined input-and-output relationship, and a method referred to 
as histogram equalization in which the level conversion function is 
adaptively changed according to frequency distribution of pixel levels 
have been proposed. 
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As the high frequency component emphasis method, a method 
referred to as an unsharp mask has been proposed that performs a 
so-called edge enhancement in which an edge is extracted from the image 
and the extracted edge is emphasized. 

5 

In the contrast emphasis method, however, there is a problem in 
that only some bright regions within all the dynamic range (a difference 
between the maximum level and the minimum level) of the image can be 
increased in contrast. In addition, there is another problem in that the 

10 contrast is instead reduced in the brightest part and the darkest part of 
an image in the case of the tone curve adjustment, and near a 
luminescence region with low frequency distribution in the case of the 
histogram equalization. Furthermore, in the high frequency component 
emphasis method there is another problem in that only the contrast of the 

15 high frequency component of the image is emphasized, whereby the 
portion near the edges of the image is unnaturally emphasized, and 
deterioration of image quality is unavoidable. 

Then, there is a conventional method in which, in a situation 
20 where edges having a steep change in a pixel value among input image 
data are saved, portions other than the edges are amplified with an image 
signal processing apparatus constructed as shown in FIG. 1, to thereby 
emphasize the portions other than the edges (for example, Japanese 
Patent Application Publication (KOKAI) No. 2001-298621). 

25 

In the image signal processing apparatus as shown in FIG. 1, an 
inputted image signal is inputted into an s filter 1 and a subtraction unit 
2. The 8 filter 1 receives, as an input, an image signal slightly changing 
on both sides of a steep edge as shown in FIG. 2A, converts it into an 
30 image signal in which only edges as shown in FIG. 2B are extracted, and 
which is outputted to the subtraction unit 2 and an adder 4. 
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Particular processing of the s filter 1 will be described with 
reference to FIGS. 3 and 4. The e filter 1 determines each pixel of the 
input image to be a pixel of attention C one by one. As shown in FIG. 3, 
5 taps are set up, including a plurality of neighbouring pixels (in this case 
six pixels L3, L2, LI, Rl, R2, R3) which are horizontally successive and 
centered around the pixel of attention C. As shown in the following 
expression (l), pixel values of the pixel of attention C and the plurality of 
neighbouring pixels are subjected to weighted averaging by means of the 
10 tap coefficients (for example, {l, 2, 3, 4, 3, 2, l} ), and outputted as a 
conversion result C* corresponding to the pixel of attention C. 



C = (1L3 + 2 L2 + 3 L1 + 4 C + 3-R1 + 2-R2 + 1R3)/16 ... (l) 



15 However, as shown in FIG. 4, a neighbouring pixel (in the case of 

FIG. 4, neighbouring pixels R2 and R3) having a difference, which is 
larger than a predetermined threshold value e, between its pixel value 
and the pixel of attention C is calculated by replacing it by that of the 
pixel of attention C. That is, in the case of FIG. 4, the following 

20 expression (2) is calculated. 



C' = (LL3 + 2 L2 + 3L1 + 4 C + 3 R1 + 2 C + 1C)/16 ... (2) 



Now, returning to FIG. 1, the subtraction unit 2 subtracts the 
25 image signal inputted from the e filter 1, from the image signal (the same 
as the input to the e filter 1) inputted from the preceding stage, extracts 
the image signal slightly changing other than that of the edge, and 
outputs it to an amplification unit 3. The amplification unit 3 amplifies 
the output of the subtraction unit 2, and outputs it to the adder 4. The 
30 adder 4 adds the image signal in which parts other than the edge 
outputted from the amplification unit 3 are amplified, to the image signal 
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in which only the edge inputted from the 8 filter 1 is extracted. This 
produced total is the image signal in which the parts other than the edge 
are amplified in the situation where the steep edge is held. 

5 Incidentally, in the s filter 1 of the image signal processing 

apparatus as shown in FIG. 1, when an image signal having a larger edge 
size than the predetermined threshold value s is inputted as shown in 
FIG. 5 for example, the image signal after conversion is such that the 
phase is shifted to the left-hand side, as shown in FIG. 6. That is, there 
10 is a problem in that the edge with the steep change in pixel value is not 
held correctly, and an image quality may deteriorate. 

DISCLOSURE OF THE INVENTION 

The present invention is invented in view of such situations, and 
15 aims to smooth portions except an edge while the edge whose change in 
pixel value is steep is held correctly. 

A signal processing apparatus of the present invention includes* 
designation means for designating continuously arranged signals as a 

20 signal of attention one by one; determination means for determining a 
predetermined number of signals preceding the signal of attention 
designated by the designation means, and a predetermined number of 
signals following the signal of attention, to be neighbouring signals* 
weight average means for averaging by weight the signal of attention and 

25 the plurality of neighbouring signals! flag setting means for calculating a 
difference in levels between the signal of attention and a neighbouring 
signal, judging whether or not the difference is larger than a 
predetermined threshold value, and raising flags for the neighbouring 
signal and a neighbouring signal which are arranged symmetrically with 

30 respect to the signal of attention, when the difference is judged to be 
larger than the predetermined threshold value; and control means for 
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controlling and causing the weighted average means to average by weight, 
using the signal of attention instead of the neighbouring signal for which 
the flag is raised. 

5 On the basis of the position of the signal of attention, the 

above-mentioned flag setting means can raise a flag also for the 
neighbouring signal arranged away from the neighbouring signal for 
which the flag has been raised. 

10 The above-mentioned signal can be the pixel value of the pixel 

which constitutes the image. 

A signal processing method of the present invention includes- a 
designation step of designating continuously arranged signals as a signal 

15 of attention one by one! a determination step of determining a 
predetermined number of signals preceding the signal of attention 
designated by way of the designation step, and a predetermined number 
of signals following the signal of attention, to be neighbouring signals; a 
weight average step of averaging by weight the signal of attention and 

20 the plurality of neighbouring signals! a flag setting step of calculating a 
difference in levels between the signal of attention and a neighbouring 
signal, judging whether or not the difference is larger than a 
predetermined threshold value, and raising flags for the neighbouring 
signal and a neighbouring signal which are arranged symmetrically with 

25 respect to the signal of attention, when the difference is judged to be 
larger than the predetermined threshold value; and a control step of 
controlling and causing a process in the weighted average step to average 
by weight, using the signal of attention instead of the neighbouring signal 
for which the flag is raised. 

30 

A program for a recording medium in accordance with the present 
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invention includes : a designation step of designating continuously- 
arranged signals as a signal of attention one by one; a determination step 
of determining a predetermined number of signals preceding the signal of 
attention designated by way of the designation step, and a predetermined 
5 number of signals following the signal of attention, to be neighbouring 
signals! a weight average step of averaging by weight the signal of 
attention and the plurality of neighbouring signals; a flag setting step of 
calculating a difference in levels between the signal of attention and a 
neighbouring signal, judging whether or not the difference is larger than 

10 a predetermined threshold value, and raising flags for the neighbouring 
signal and a neighbouring signal which are arranged symmetrically with 
respect to the signal of attention, when the difference is judged to be 
larger than the predetermined threshold value; and a control step of 
controlling and causing a process in the weighted average step to average 

15 by weight, using the signal of attention instead of the neighbouring signal 
for which the flag is raised. 

A program in accordance with the present invention causes a 
computer to implement processes including- a designation step of 

20 designating continuously arranged signals as a signal of attention one by 
one; a determination step of determining a predetermined number of 
signals preceding the signal of attention designated by way of the 
designation step, and a predetermined number of signals following the 
signal of attention, to be neighbouring signals; a weight average step of 

25 averaging by weight the signal of attention and the plurality of 
neighbouring signals; a flag setting step of calculating a difference in 
levels between the signal of attention and a neighbouring signal, judging 
whether or not the difference is larger than a predetermined threshold 
value, and raising flags for the neighbouring signal and a neighbouring 

30 signal which are arranged symmetrically with respect to the signal of 
attention, when the difference is judged to be larger than the 
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predetermined threshold value; and a control step of controlling and 
causing a process in the weighted average step to average by weight, 
using the signal of attention instead of the neighbouring signal for which 
the flag is raised. 

5 

In the signal processing apparatus and the method, and the 
program in accordance with the present invention, the difference in levels 
between the signal of attention and the neighbouring signal is calculated, 
and it is judged whether or not the difference is larger than the 

10 predetermined threshold value. When the difference is judged to be 
larger than the predetermined threshold value, flags are raised for the 
neighbouring signal and a neighbouring signal which are arranged 
symmetrically with respect to the signal of attention. In the case of 
averaging by weight the signal of attention and the plurality of 

15 neighbouring signals, the signal of attention is used instead of the 
neighbouring signals for which the flags are raised. 



BRIEF DESCRIPTION OF THE DRAWINGS 



20 FIG. 1 is a block diagram showing an example of a structure of an 

image signal processing apparatus which emphasizes portions except an 
edge while a steep edge in an image is saved; 

FIG. 2A is a graph showing an image signal inputted into and an 
image signal outputted from an e filter of FIG. 1; 
25 FIG. 2B is a graph showing an image signal inputted into and an 

image signal outputted from the e filter of FIG. l; 

FIG. 3 is a diagram showing an example of taps used for the 8 
filter of FIG. l; 

FIG. 4 is a chart for explaining operation of the 8 filter of FIG. i; 
30 FIG. 5 is a chart showing an example of the image signal inputted 

into the 8 filter; 
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FIG. 6 is a chart showing an example of the image signal 
outputted from the e filter, corresponding to the image signal as shown in 
FIG. 5; 

FIG. 7 is a block diagram showing an example of a structure of a 
5 nonlinear filter to which the present invention is applied; 

FIG. 8 is a flow chart for explaining a first filtering process by 
means of the nonlinear filter of FIG. 7; 

FIG. 9 is a chart showing the image signal acquired by way of the 
first filtering process with the nonlinear filter, corresponding to the image 
10 signal as shown in FIG. 5; 

FIG. 10 is a chart showing an example of the image signal 
inputted into the nonlinear filter; 

FIG. 11 is a chart showing an image signal acquired by way of the 
first filtering process with the nonlinear filter, corresponding to the image 
15 signal as shown in FIG. 10; 

FIG. 12 is a flow chart for explaining a second filtering process by 
means of the nonlinear filter of FIG. 7; 

FIG. 13 is a chart showing an image signal acquired by the second 
filtering process with the nonlinear filter, corresponding to the image 
20 signal as shown in FIG. 10; and 

FIG. 14 is a block diagram showing an example of a structure of a 
general-purpose personal computer. 

BEST MODE FOR CARRYING OUT THE INVENTION 
25 FIG. 3 shows an example of a structure of a nonlinear filter which 

is an embodiment of the present invention. This nonlinear filter 11 is 
used by replacing the 8 filter 1 of the image signal processing apparatus 
as shown in FIG. 1, and it is constituted by a control signal generating 
unit 12 which generates a control signal based on an inputted image 
30 signal, and an LPF (Low Pass Filter) 13 which performs a filtering 
operation according to a control signal generated by the control signal 
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generating unit 12. 

A first filtering process by means of the nonlinear filter 11 will be 
described with reference to a flow chart of FIG. 8, referring to a case 
5 where 7-pixel taps, containing the pixel of attention C as shown in FIG. 3, 
are used as an example. 

In step SI, the control signal generating unit 12 determines each 
of pixels in order of raster which constitute the inputted image signal, to 

10 be the pixel of attention C one by one. In step S2, the control signal 
generating unit 12 determines the neighbouring pixels L3, L2, LI, Rl, R2, 
and R3 which adjoin in the horizontal direction with respect to the pixel 
of attention C. In step S3, the control signal generating unit 12 
calculates a difference in pixel values between the pixel of attention C 

15 and each of the neighbouring pixels L3, L2, LI, Rl, R2, and R3. 

In step S4, the control signal generating unit 12 judges whether 
or not the difference calculated by way of the process in step S3 is larger 
than the predetermined threshold value s. Flags are raised for the 
20 neighbouring pixel judged to have the difference larger than the 
predetermined threshold value 8, and a neighbouring pixel which are 
arranged symmetrically with respect to the pixel of attention C. 

For example, when it is judged that the difference in pixel values 
25 between the pixel of attention C and the neighbouring pixel LI is larger 
than the predetermined threshold value e, flags are raised for the 
neighbouring pixels LI and Rl. Similarly, when it is judged that the 
difference in pixel values between the pixel of attention C and the 
neighbouring pixel R2 is larger than the predetermined threshold value s, 
30 flags are raised for the neighbouring pixels R2 and L2. 
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Furthermore, in step S4, the control signal generating unit 12 
outputs a signal indicative of whether or not there is a flag for the 
neighbouring pixels L3, L2, LI, Rl, R2, and R3, to the LPF 13 as a control 
signal. 

5 

In step S5, the LPF 13 averages by weight the pixel of attention C 
and the neighbouring pixels L3, L2, LI, Rl, R2, and R3 by using 
expression (l), to be outputted as the conversion result C corresponding 
to the pixel of attention C. However, with respect to the neighbouring 
10 pixel for which the flag is raised, the pixel value is replaced by that of the 
pixel of attention C, and is calculated. 

For example, when the flags are raised for the neighbouring 
pixels L2 and R2, the following expression (3) is operated. 

15 

C = (1L3 + 2 C + 3 L1 + 4 C + 3R1 + 2 C + 1R3)/16 ... (3) 

Now, the description of the first filtering process is completed. 
According to this first filtering process, as shown in FIG. 5, for example, 
20 when the image signal with the edge larger in size than the 
predetermined threshold value e is inputted, the image signal after 
conversion is, as shown in FIG. 9, not shifted in phase and the edge with 
the steep change in pixel value is held correctly. 

25 However, as shown in FIG. 10, for example, when an image signal 

with the edge larger in size than the predetermined threshold value s is 
inputted, the edge with the steep change in pixel value, with respect to 
the image signal after conversion, may not be held correctly even by way 
of the first filtering process, as shown in FIG. 11. 

30 



Then, the nonlinear filter 11 is arranged to perform a second 



S04P0571 



filtering process as will be described below. The second filtering process 
will be described with reference to a flow chart of FIG. 12, referring to a 
case where the 7 -pixel taps, containing the pixel of attention C as shown 
in FIG. 3, are used as an example. 

5 

In step Sll, the control signal generating unit 12 determines each 
of the pixels in order of raster which constitute the inputted image signal 
to be the pixel of attention C one by one. In step S12, the control signal 
generating unit 12 determines the neighbouring pixels L3, L2, LI, Rl, R2, 
10 and R3 which adjoin in the horizontal direction with respect to the pixel 
of attention C. In step S13, the control signal generating unit 12 
calculates a difference in pixel values between the pixel of attention C 
and each of the neighbouring pixels L3, L2, LI, Rl, R2, and R3. 

15 In step S14, the control signal generating unit 12 judges whether 

or not the difference calculated by way of the process in step S13 is larger 
than the predetermined threshold value 8. Flags are raised for the 
neighbouring pixel judged to have the difference larger than the 
predetermined threshold value e, and a neighbouring pixel, which are 

20 arranged symmetrically with respect to the pixel of attention C. 
Furthermore, the control signal generating unit 12 also raises a flag for a 
neighbouring pixel away, in view of the pixel of attention C, from the 
right and left neighbouring pixels which are symmetrical with respect to 
the pixel of attention C. 

25 

For example, when it is judged that the difference in pixel values 
between the pixel of attention C and the neighbouring pixel L2 is larger 
than the predetermined threshold value s, flags are raised for the 
neighbouring pixels L2 and R2. Furthermore, flags are also raised for 
30 the neighbouring pixels L3 and R3 which are away from the neighbouring 
pixels L2 and R2 in view of the pixel of attention C. 
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Furthermore, in step S14, the control signal generating unit 12 
outputs a signal indicative of whether or not there is a flag for the 
neighbouring pixels L3, L2, LI, Rl, R2, and R3, to the LPF 13 as a control 
5 signal. 

In step S15, the LPF 13 averages by weight the pixel of attention 
C and the neighbouring pixels L3, L2, Ll, Rl, R2, and R3 by using 
expression (l), to be outputted as the conversion result C f corresponding 
10 to the pixel of attention C. However, with respect to the neighbouring 
pixel for which the flag is raised, the pixel value is replaced by that of the 
pixel of attention C, and is calculated. 

For example, when the flags are raised for the neighbouring 
15 pixels L3, L2, R2, and R3 the following expression (4) is operated. 

C = (1C + 2-C + 3-L1 + 4 C + 3 R1 + 2 C + 1C)/16 ... (4) 

Now, the description of the second filtering process is completed. 
20 Also according to this second filtering process, as shown in FIG. 5, for 
example, when the image signal with the edge larger in size than the 
predetermined threshold value s is inputted, the image signal after 
conversion is not shifted in phase as shown in FIG. 9, and the edge with 
the steep change in pixel value is held correctly. 

25 

Further, for example, as shown in FIG. 10, even when the image 
signal with the edge larger in size than the predetermined threshold 
value s is inputted, the image signal after conversion is not shifted in 
phase as shown in FIG. 13, and the edge with the steep change in pixel 
30 value is held correctly. 
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In addition, the present invention can be applied to any device 
handling an image signal, such as a video camera, a digital still camera, a 
printer, a display, a computer, etc. 

5 For example, when it is applied to a computer which performs 

image processing, a high-definition contrast correction image can be 
obtained maintaining a dynamic range in case of correcting the image 
contrast. When the images obtained under different lighting conditions 
are composed, only the difference of each contrast component can be 
10 corrected and a natural composite image can be generated. 

Incidentally, although a series of processes as mentioned above 
can be performed by means of hardware, they can also be performed by 
way of software. When performing the series of processes by way of 
15 software, a program constituting the software is installed from a 
recording medium in a computer which is built into dedicated hardware, 
or a general-purpose personal computer which has various types of 
programs installed therein and is capable of performing various types of 
functions as shown in FIG. 14, for example. 

20 

This personal computer 50 contains a CPU (Central Processing 
Unit) 51 therein. The CPU 51 is connected through a bus 54 to an 
input/output interface 55. The bus 54 is connected with a ROM (Read 
Only Memory) 52 and a RAM (Random Access Memory) 53. 

25 

The input/output interface 55 is connected with an input unit 56 
made of an input device, such as a keyboard, a mouse, a remote controller, 
etc. through which a user inputs an operation command, an output unit 
57 for outputting a composite picture signal to a display, a storage unit 58 
30 made of a hard disk drive etc. for storing a program and various data, and 
a communications unit 59 made of a modem, a LAN (Local Area Network) 



14 S04P0571 



adapter, etc., for performing a communications process through a network 
represented by the Internet. Further, it is connected with a drive 60 for 
reading/writing data from/to a recording media, such as a magnetic disk 
61 (including a flexible disk), an optical disc 62 (including a CD-ROM 
5 (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a 
magneto-optical disc 63 (including an MD (Mini Disc)), a semiconductor 
memory 64, etc. 

The program which causes the CPU 51 to carry out the 
10 above-mentioned first or second filtering process is supplied to the 
personal computer in the situation where it is stored in the magnetic disk 
61 through the semiconductor memory 64, and it is read by the drive 60 
and installed in the hard disk drive which is built in the storage unit 58. 
Alternatively, it is envisaged that it may be supplied through the network. 
15 The program installed in the storage unit 58 is loaded from the storage 
unit 58 to the RAM 53 according to instructions from the CPU 51 
corresponding to the command, from the user, inputted into the input 
unit 56, and is executed. 

20 In addition, in the present specification, steps which describe each 

flowchart include processes performed in chronological order according to 
the description but also processes carried out in parallel or individually, 
even if they are not necessarily processed in chronological order. 

25 INDUSTRIAL APPLICABILITY 

As described above, according to the present invention, the 
portions except the edge can be smoothed while the edge whose change in 
pixel value is steep is held correctly. 



